I'm convinced that the nut's very nutritiousness is to blame for the dish's poor reputation. They're so dense that a loaf made primarily from nuts would be more suitable for slicing into energy bars and selling to mountaineering supply shops - hence the main bulk of a nut roast is generally some form of carbohydrate, intended to lighten the load.
Source: wiktionary